Inspection system, inspection method, program, and storage medium

ABSTRACT

An inspection system includes an acquisition unit and a determination unit. The acquisition unit acquires an image representing a surface of an object. The determination unit performs color determination processing. The color determination processing is performed to determine a color of the surface of the object based on a plurality of conditions of reflection. The plurality of conditions of reflection are obtained from the image representing the surface of the object as acquired by the acquisition unit, and have a specular reflection component and a diffuse reflection component at respectively different ratios on the surface of the object.

CROSS-REFERENCE OF RELATED APPLICATIONS

This application is a Continuation of U.S. patent application Ser. No.16/770,537, filed on Jun. 5, 2020, which is the U.S. National Phaseunder 35 U.S.C. § 371 of International Patent Application No.PCT/JP2018/045094, filed on Dec. 7, 2018, which is based on and claimspriority of U.S. Provisional Patent Application No. 62/699,935, filed onJul. 18, 2018, U.S. Provisional Patent Application No. 62/699,942, filedon Jul. 18, 2018, and U.S. Provisional Patent Application No.62/596,247, filed on Dec. 8, 2017, the entire disclosures of whichapplications are incorporated by reference herein.

TECHNICAL FIELD

The present disclosure generally relates to an inspection system, aninspection method, a program, and a storage medium, and moreparticularly relates to an inspection system, inspection method,program, and storage medium, all of which are configured or designed tomake a decision about the surface of an object using an image.

BACKGROUND ART

Patent Literature 1 discloses a coloring inspection device. The coloringinspection device of Patent Literature 1 includes: a camera having threespectral sensitivities that have been linearly transformed into valuesequivalent to a CIEXYZ color matching function; an arithmetic-logic unit(processor) for calculating and acquiring coloring data by transformingan image having the three spectral sensitivities and captured by thecamera into tristimulus values X, Y, and Z in the CIEXYZ color system;and a lighting unit for irradiating an automobile, which is an exemplaryobject of measurement, with light. The coloring inspection devicecarries out color inspection by calculating an index of colordistribution matching indicating the ratio of overlap between two xyzchromaticity histograms of an object under test and a reference object.

CITATION LIST Patent Literature

Patent Literature 1: JP 2015-155892 A

SUMMARY OF INVENTION

It is an object of the present disclosure to provide an inspectionsystem, an inspection method, a program, and a storage medium, all ofwhich are configured or designed to improve the accuracy of surfacecolor determination of a given object.

An inspection system according to an aspect of the present disclosureincludes an acquisition unit and a determination unit. The acquisitionunit acquires an image representing a surface of an object. Thedetermination unit performs color determination processing. The colordetermination processing is performed to determine a color of thesurface of the object based on a plurality of conditions of reflection.The plurality of conditions of reflection are obtained from the imagerepresenting the surface of the object as acquired by the acquisitionunit, and have a specular reflection component and a diffuse reflectioncomponent at respectively different ratios on the surface of the object.

An inspection method according to another aspect of the presentdisclosure includes an acquisition step and a determination step. Theacquisition step includes acquiring an image representing a surface ofan object. The determination step includes performing colordetermination processing. The color determination processing isperformed to determine a color of the surface of the object based on aplurality of conditions of reflection. The plurality of conditions ofreflection are obtained from the image representing the surface of theobject, and have a specular reflection component and a diffusereflection component at respectively different ratios on the surface ofthe object.

A program according to still another aspect of the present disclosure isdesigned to cause one or more processors to execute the inspectionmethod described above.

A storage medium according to yet another aspect of the presentdisclosure is a computer-readable non-transitory storage medium storingthe program described above thereon.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram illustrating an inspection system according toan exemplary embodiment;

FIG. 2 illustrates how the inspection system works;

FIG. 3 is a flowchart showing the procedure of setting processing to beperformed by the inspection system;

FIG. 4 is a flowchart showing the procedure of color determinationprocessing to be performed by the inspection system;

FIG. 5 illustrates a first separate image;

FIG. 6 illustrates a second separate image;

FIG. 7 illustrates how the inspection system performs a type ofseparation processing;

FIG. 8 illustrates how the inspection system performs another type ofseparation processing;

FIG. 9 illustrates how the inspection system performs synthesisprocessing in one way;

FIG. 10 illustrates how the inspection system performs synthesisprocessing in another way;

FIG. 11 illustrates how the inspection system performs color controlprocessing;

FIG. 12 is a flowchart showing the procedure of painting processingincluding painting determination processing to be performed by theinspection system;

FIG. 13 illustrates how the inspection system performs a type ofpainting processing;

FIG. 14 illustrates how the inspection system performs another type ofpainting processing;

FIG. 15 is a flowchart showing the procedure of texture determinationprocessing to be performed by the inspection system;

FIG. 16 illustrates how the inspection system performs texturedetermination processing; and

FIG. 17 illustrates an image capturing system.

DESCRIPTION OF EMBODIMENTS 1. Embodiments

1.1 Overview

FIG. 1 illustrates an inspection system 1 according to an exemplaryembodiment. The inspection system 1 includes an acquisition unit F11 anda determination unit F13. The acquisition unit F11 acquires an imagerepresenting a surface of an object 100. The determination unit F13performs color determination processing. The color determinationprocessing is performed to determine a color of the surface of theobject 100 based on a plurality of conditions of reflection. Theplurality of conditions of reflection are obtained from the imagerepresenting the surface of the object 100 as acquired by theacquisition unit F11, and have a specular reflection component and adiffuse reflection component at respectively different ratios on thesurface of the object 100.

The inspection system 1 determines the color of the surface of theobject 100 from multiple viewpoints, not a single viewpoint. Moreparticularly, the inspection system 1 determines the color of thesurface of the object 100 based on a plurality of conditions ofreflection having a specular reflection component and a diffusereflection component at respectively different ratios on the surface ofthe object 100. Specifically, the specular reflection componentrepresents the surface condition of the object 100 more faithfully thanthe diffuse reflection component does. On the other hand, the diffusereflection component represents the color of the surface itself of theobject 100 more accuracy than the specular reflection component does.This allows the color of the surface of the object 100 to be determinedwith not only the color of the surface itself of the object 100 but alsothe surface condition of the object 100 taken into account.Consequently, this inspection system 1 improves the accuracy of surfacecolor determination of the object 100.

1.2 Details

Next, the inspection system 1 will be described in further detail withreference to the accompanying drawings. This inspection system 1 is asystem for subjecting an object 100 to some type of inspection. Forexample, the inspection system 1 may serve as a coloring inspectiondevice. In this embodiment, the inspection is supposed to be performedby the inspection system 1 with respect to the color, paintingcondition, and texture of the surface of the object 100. In addition,the inspection system 1 may also paint the object 100. The inspectionsystem 1 may paint the object 100 based on a result of the inspection,thereby turning the object 100 into an object 100 painted in any desiredcolor(s).

The object 100 may be any type of object having a surface. Specifically,in this embodiment, the object 100 is an automobile. Specifically, thesurface of the object 100 is an outer surface of the vehicle body of theautomobile. However, this is only an example and should not be construedas limiting. That is to say, the object 100 does not have to be anautomobile but may also be any other type of moving vehicle or may evenbe an object other than moving vehicles. Examples of the moving vehiclesinclude bicycles, motorcycles, railway trains, drones, aircrafts,construction machines, ships, and boats. The object 100 may even be anelectric device, a piece of tableware, a container, a piece offurniture, clothes, or a building material. In short, the object 100 maybe any type of object having a surface. In particular, the inspectionsystem 1 according to this embodiment is effectively applicable to anyobject to be painted.

As shown in FIG. 1, the inspection system 1 includes a determinationsystem 10, a lighting system 20, an image capturing system 30, and apainting system 40.

The lighting system 20 is a system for irradiating the surface of theobject 100 with light. As shown in FIG. 2, the lighting system 20includes a plurality of (e.g., four in the example shown in FIG. 2)lamps 21 for irradiating the object 100 with light. The lamps 21 may beLED lamps, for example. Each of these lamps 21 radiates white light.Note that this lighting system 20 may include any other number of lamps21, which do not have to be LED lamps but may also be any other type oflight sources. The light emitted from the lamps 21 does not have to bewhite light, either. The color of the light emitted from the lamps 21may be determined appropriately with the color of the object 100 and thecolors detectable for the image capturing system 30 taken into account.Optionally, the wavelength of the light radiated by the lighting system20 may be changeable. In this embodiment, the inspection system 1includes the lighting system 20. However, if the color of the object 100is determinable even without the lighting system 20, the lighting system20 may be omitted.

The image capturing system 30 is a system for generating an image(digital image) representing the surface of the object 100. In thisembodiment, the image capturing system 30 generates an imagerepresenting the surface of the object 100 by capturing an image of thesurface of the object 100 being irradiated by the lighting system 20.The image capturing system 30 includes a plurality of cameras, each ofwhich includes one or more image sensors. Optionally, each camera mayinclude one or more line sensors.

In this embodiment, the plurality of cameras of the image capturingsystem 30 includes four (first) cameras 31 (311-314) and one (second)camera 32 as shown in FIG. 2. In this case, the first cameras 31 and thesecond camera 32 may be displaced with respect to the object 100. Also,the second camera 32 suitably has a wider angle of view than the firstcameras 31 and is suitably able to generate an image representing theobject 100 in its entirety. For example, the second camera 32 may belocated to overlook the object 100 to generate an image representing abird's-eye view of the object 100.

The painting system 40 is a system for painting the surface of theobject 100. As shown in FIG. 2, the painting system 40 includes aplurality of painting units (e.g., two painting units 41, 42). In thisembodiment, the painting units 41, 42 are both implemented as paintingrobots. The painting units 41, 42 are displaceable with respect to theobject 100. The painting robots may have a known configuration, anddetailed description thereof will be omitted herein. Note that thepainting system 40 needs to include at least one painting unit, andtherefore, the number of the painting units provided does not have to betwo.

The determination system 10 includes an input/output unit 11, a storageunit 12, and a processing unit 13 as shown in FIG. 1. The determinationsystem 10 may be implemented as a computer system. The computer systemmay include one or more processors, one or more connectors, one or morecommunications devices, and one or more memories, for example.

The input/output unit 11 is an interface through which information isinput and output from one of the lighting system 20, the image capturingsystem 30, or the painting system 40 to another. In this embodiment, theinput/output unit 11 is connected to be communicable with the lightingsystem 20, the image capturing system 30, and the painting system 40.The input/output unit 11 includes one or more input/output devices, anduses one or more input/output interfaces.

The storage unit 12 is used to store information to be used by theprocessing unit 13. The storage unit 12 includes one or more storagedevices, which may be a random access memory (RAM) and/or anelectrically erasable programmable read-only memory (EEPROM). Thestorage unit 12 stores sample data to be used in the color determinationprocessing. The sample data includes information about a target color ofthe surface of the object 100 (i.e., color data as samples). Theinformation about the target color of the surface of the object 100 maybe provided as reflectance values on a wavelength basis (see FIG. 11).For example, the information about the target color of the surface ofthe object 100 may be provided as reflectance values falling within thewavelength range from 380 nm to 780 nm. That is to say, the sample datahas an aspect as digital color samples. In addition, the sample data mayalso include at least one of information about the shape of the object100 (information about the shape of the object of shooting) orinformation about a condition for capturing an image of the object 100.The condition for capturing an image of the object 100 may includerelative positional relation between the object 100, the lighting system20, and the image capturing system 30 (i.e., information about therelative positions of the object of shooting, the lighting, and thecameras). In other words, the condition for capturing an image of theobject 100 may include information about the lighting provided by thelighting system 20 (hereinafter referred to as “lighting information”).For example, the sample data may be an image representing a whole of thesurface of the object 100 that has been captured under a predeterminedimage capturing condition. Besides, the storage unit 12 may furtherstore standard information for use in the painting determinationprocessing. The standard information represents a target surfacepainting condition of the object 100. In other words, the standardinformation indicates in what condition the object 100 should bepainted.

The processing unit 13 may be implemented as one or more processors (ormicroprocessors). That is to say, the one or more processors perform thefunction of the processing unit 13 by executing one or more programs(computer programs) stored in one or more memories. The one or moreprograms may be stored in advance in the one or more memories.Alternatively, the one or more programs may also be downloaded via atelecommunications line such as the Internet or distributed after havingbeen stored in a non-transitory storage medium such as a memory card.

The processing unit 13 performs setting processing (see FIG. 3), colordetermination processing (see FIG. 4), painting determination processing(see FIG. 12), and texture determination processing (see FIG. 15). Theprocessing unit 13 includes the acquisition unit F11, a separation unitF12, and the determination unit F13 as shown in FIG. 1. Note that theacquisition unit F11, separation unit F12, and determination unit F13 donot have a substantive hardware configuration but are actuallyrespective functions to be performed by the processing unit 13.

The acquisition unit F11 acquires an image representing the surface ofthe object 100 (see FIGS. 9 and 10). In this embodiment, the acquisitionunit F11 acquires an image representing the surface of the object 100from the image capturing system 30. That is to say, the acquisition unitF11 receives the image from the image capturing system 30 via theinput/output unit 11. It depends on the image capturing condition forthe image capturing system 30 what image the acquisition unit F11acquires from the image capturing system 30.

The separation unit F12 performs separation processing. The separationprocessing is processing for obtaining, from the image representing thesurface of the object 100 as acquired by the acquisition unit F11, aplurality of conditions of reflection having a specular reflectioncomponent and a diffuse reflection component at respectively differentratios on the surface of the object 100. When performing the separationprocessing, the separation unit F12 obtains, as images representing theplurality of conditions of reflection having a specular reflectioncomponent and a diffuse reflection component at respectively differentratios on the surface of the object 100, a plurality of separate imagesfrom the image acquired by the acquisition unit F11. The plurality ofseparate images represents respective surface images of the object 100and are images having a specular reflection component and a diffusereflection component at respectively different ratios on the surface ofthe object. In this embodiment, the plurality of separate images are afirst separate image P10 (see FIG. 5) and a second separate image (seeFIG. 6). However, the number of the separate images does not have to betwo but may also be three or more.

As used herein, the “plurality of conditions of reflection” includes thecondition of reflection of light from the surface of the object 100.When the surface of the object 100 is viewed from a different direction,the ratio of the specular reflection component and the diffusereflection component is variable. Therefore, it can be said that theplurality of conditions of reflection are the conditions of the surfaceof the object 100 as viewed from multiple different viewpoints. Imagesrepresenting multiple different conditions of reflection from thesurface of the object 100 may be obtained by capturing images of thesurface of the same object 100 using a plurality of cameras set up atmultiple different positions. When images of the surface of the object100 are captured with a plurality of cameras set up at multipledifferent positions, each of the images thus captured may include both aspecular reflection component and a diffuse reflection component.Nevertheless, either only the specular reflection component or only thediffuse reflection component may be extracted from the given imagethrough arithmetic processing. As shown in FIG. 7, the specularreflection component has a peak around a region on an image capturingplane I where the angle of incidence of the light from a light source Lonto a surface S is equal to the angle of reflection of the light fromthe surface S. Therefore, the specular reflection component is dominantin a region on the image capturing plane I corresponding to apredetermined range (θ±φ) centered around the angle of reflection θ.Meanwhile, the diffuse reflection component is dominant in the remainingregion on the image capturing plane I excluding the region correspondingto the predetermined range (θ±φ). Therefore, on the image capturingplane I, the intensity of the reflected light is given as a syntheticvalue of the specular reflection component and the diffuse reflectioncomponent. In addition, the specular reflection component and thediffuse reflection component may be estimated based on a statisticalmodel. Furthermore, the specular reflection component and the diffusereflection component have different intensity distributions, andtherefore, are separable from each other based on the grayscale values(luminance values) of the image as shown in FIG. 8.

In this embodiment, the separation unit F12 extracts the first separateimage P10 (see FIG. 5) and the second separate image (see FIG. 6) fromthe image acquired by the acquisition unit F11. Therefore, the pluralityof conditions of reflection may include a condition with only thespecular reflection component and a condition with only the diffusereflection component. The first separate image P10 represents acondition in which the surface of the object 100 has only the specularreflection component. The second separate image P20 represents acondition in which the surface of the object 100 has only the diffusereflection component. In this case, the specular reflection componentrepresents the surface condition of the object 100 more faithfully thanthe diffuse reflection component does. On the other hand, the diffusereflection component represents the color of the surface itself of theobject 100 more accurately than the specular reflection component does.

The determination unit F13 performs the setting processing (see FIG. 3),the color determination processing (see FIG. 4), the painting processing(see FIG. 12), and the texture determination processing (see FIG. 15).

(Setting Processing)

The setting processing is pre-processing to be performed before thecolor determination processing. The setting processing includes enteringsettings into the image capturing system 30. To enter settings into theimage capturing system 30, the determination unit F13 determines animage capturing condition for the image capturing system 30. The imagecapturing condition defines the operating conditions of a plurality ofcameras (in particular, the plurality of first cameras 31) of the imagecapturing system 30. The operating conditions may include theirpositions with respect to the surface of the object 100, an imagecapturing direction with respect to the surface of the object 100, theangle of view (sight), and a zoom power (zooming). In this embodiment,the four first cameras 311-314 of the image capturing system 30 generatepartial images P31-P34 representing respective parts of the surface ofthe object 100 as shown in FIGS. 9 and 10. In this embodiment, theplurality of partial images P31-P34 are generated by a plurality ofcameras 31 having respectively different image capturing directions withrespect to the object 100. Thus, to produce an image (overall image) P30representing the whole of the surface of the object 100 based on thesepartial images P31-P34 generated by the four first cameras 311-314, thedetermination unit F13 determines the image capturing condition for theimage capturing system 30 as shown in FIG. 9. The determination unit F13enters settings into the image capturing system 30 in accordance withthe image capturing condition thus determined. This allows the imagecapturing system 30 to obtain an overall image of the object 100. Notethat in this embodiment, the whole of the surface of the object 100 onlyneeds to be the entire region to be subjected to the color determinationprocessing and does not have to be the whole surface of the object 100in a practical sense.

Next, the procedure of the setting processing will be described withreference to the flowchart shown in FIG. 3.

First of all, the determination unit F13 makes the acquisition unit F11acquire the plurality of partial images P31-P34 of the object 100 fromthe image capturing system 30 (in S11). A first separate image and asecond separate image are obtained (extracted) by the separation unitF12 from each of the plurality of partial images P31-P34 acquired by theacquisition unit F11 (in S12). The determination unit F13 synthesizestogether the respective first separate images of the plurality of thepartial images P31-P34 to generate a first synthetic image. In addition,the determination unit F13 also synthesizes together the respectivesecond separate images of the plurality of the partial images P31-P34 togenerate a second synthetic image (in S13). The determination unit F13determines an image capturing condition of the image capturing system 30(in terms of the operating conditions of the four first cameras 311-314)such that continuity of the object 100 is maintained with respect toeach of the first synthetic image and the second synthetic image (inS14). As used herein, if the continuity of the object 100 (i.e., thecontinuity of gradation of the object of shooting) is maintained, thenit means that the shape of the object 100 is expressed accurately in thesynthetic image generated by synthesizing together the partial images.FIG. 9 illustrates a situation where the continuity of the object 100 ismaintained. On the other hand, FIG. 10 illustrates a situation where thecontinuity of the object 100 is not maintained. Note that to make thedescription more easily understandable, FIGS. 9 and 10 illustrate anexample in which a synthetic image has been generated based on thepartial images P31-P34, instead of the respective first separate imagesor second separate images. The determination unit F13 may determine theimage capturing condition by reference to the information about theshape of the object 100 included in the sample data stored in thestorage unit 12. The determination unit F13 determines the imagecapturing condition such that the shape of the object 100 in the firstseparate image and the second separate image matches the shape of theobject 100 as indicated by the information included in the sample data.In accordance with the image capturing condition thus determined, thedetermination unit F13 enters settings into the image capturing system30 (in S15). In this manner, the operating condition of the four firstcameras 311-314 is updated. This allows the image capturing system 30 toobtain the overall image of the object 100.

As can be seen, the inspection system 1 synthesizes together the images(partial images) captured by the plurality of cameras 31 to generatesynthetic images (i.e., a first synthetic image and a second syntheticimage), calculates an image capturing condition based on the syntheticimages, and then outputs the image capturing condition. Then, theinspection system 1 controls the angles of view and zooming of thecameras 31 such that the continuity of gradation of the object ofshooting (object 100) is maintained based on the synthetic image.

(Color Determination Processing)

The color determination processing is processing for determining thecolor of the surface of the object 100. More specifically, the colordetermination processing is processing for determining the color of thesurface of the object 100 based on a plurality of conditions ofreflection, having a specular reflection component and a diffusereflection component at respectively different ratios on the surface ofthe object 100. The plurality of conditions of reflection are obtainedfrom the image representing the surface of the object 100 as acquired bythe acquisition unit F11. In particular, in this color determinationprocessing, the determination unit F13 determines the color of thesurface of the object 100 based on a plurality of separate images P10and P20. In addition, in this color determination processing, thedetermination unit F13 determines the color of the surface of the object100 based on the plurality of images P10 and P20 each generated from theplurality of partial images P31-P34 and each representing the whole ofthe surface of the object 100 in each of the plurality of conditions ofreflection. Note that the color of the surface of the object 100 may bedetermined on a pixel-by-pixel basis on the images P10 and P20.

Next, the procedure of the color determination processing will bedescribed with reference to the flowchart of FIG. 4.

First of all, the determination unit F13 makes the acquisition unit F11acquire the plurality of partial images P31-P34 of the object 100 fromthe image capturing system 30 (in S21). A first separate image and asecond separate image are obtained (extracted) by the separation unitF12 from each of the plurality of partial images P31-P34 acquired by theacquisition unit F11 (in S22). The determination unit F13 synthesizestogether the respective first separate images of the plurality of thepartial images P31-P34 to generate a first synthetic image. In addition,the determination unit F13 also synthesizes together the respectivesecond separate images of the plurality of the partial images P31-P34 togenerate a second synthetic image (in S23). The determination unit F13compares the color of the surface of the object 100 expressed in each ofthe first synthetic image and the second synthetic image with theinformation about the target color of the surface of the object 100included in the sample data stored in the storage unit 12 (in S24). Theinformation about the target color of the surface of the object 100includes information about the target color of the surface for the firstsynthetic image and information about the target color of the surfacefor the second synthetic image. Thus, the determination unit F13determines the color of the object 100 with respect to each of the firstsynthetic image and the second synthetic image. For example, whenfinding the degree of matching between the color actually obtained fromthe first synthetic image and the target color included in the sampledata for the first synthetic image to be equal to or greater than aprescribed value, the determination unit F13 determines that the colorobtained from the first synthetic image should be a GO. Likewise, whenfinding the degree of matching between the color actually obtained fromthe second synthetic image and the target color included in the sampledata for the second synthetic image to be equal to or greater than aprescribed value, the determination unit F13 determines that the colorobtained from the second synthetic image should be a GO. In this manner,the determination unit F13 determines the color of the surface of theobject 100 based on an image representing the whole of the surface ofthe object 100 in each of a plurality of conditions of reflection andgenerated based on the plurality of partial images. When finding each ofthe colors obtained from the first synthetic image and the secondsynthetic image to be a GO, the determination unit F13 determines theresult of the color determination processing to be a GO (if the answeris YES in S25).

On the other hand, when finding at least one of the colors obtained fromthe first synthetic image and the second synthetic image to be a NO-GO,the determination unit F13 determines the result of the colordetermination processing to be a NO-GO (if the answer is NO in S25). Inthat case, the determination unit F13 paints the object 100 all overagain (in S26). In this re-painting processing step, the determinationunit F13 controls the painting system 40 based on the difference betweenthe color obtained from the first synthetic image and the target colorfor the first synthetic image and the difference between the colorobtained from the second synthetic image and the target color for thesecond synthetic image. That is to say, the determination unit F13controls, based on the result of the color determination processing, thepainting system 40 for painting the surface of the object 100. Thisrenders the color of the surface of the object 100 closer to the targetcolor. For example, FIG. 11 shows the relationship between a graph G10,G11 representing the color obtained from the synthetic image (namely,the first synthetic image or the second synthetic image) and a graph G20representing the target color for the synthetic image. In FIG. 11, thecurve G10 indicates the color before re-painting and G11 indicates thecolor after re-painting. This allows the inspection system 1 to modifythe color of the surface of the object 100.

(Painting Processing)

The painting processing is processing for painting (hereinafter alsoreferred to as “coating”) the object 100. The painting processing willnow be described with reference to the flowchart of FIG. 12 and FIGS. 13and 14, each of which illustrates one scene of painting.

First of all, the determination unit F13 determines the region to paint(or the region to coat) on the surface of the object 100 (in S31). Inthis embodiment, the painting system 40 includes the two painting robots41 and 42, and therefore, two regions to paint may be selected at a timefrom the surface of the object 100. Examples of the regions to paintinclude a hood, a roof, front doors, rear doors, a front bumper, a rearbumper, fenders, rear fenders, and a trunk lid. The determination unitF13 determines the region to paint using the image capturing system 30.The determination unit F13 makes the second camera 32 monitor the wholeof the surface of the object 100 to find what regions on the surface ofthe object 100 are not painted yet (unpainted regions). Thedetermination unit F13 selects the next region to paint from theunpainted regions on the surface of the object 100.

Next, the determination unit F13 carries out pre-painting check (inS32). In the pre-painting check, the determination unit F13 determines,before painting the region to paint, whether or not there is any foreignmatter in the region to be paint. The determination unit F13 controlsthe image capturing system 30 to make one of the plurality of firstcameras 31 (e.g., the first camera 312 in the example illustrated inFIG. 13) capture an image of the region to paint and acquires thecaptured image via the acquisition unit F11. The determination unit F13detects, by image processing technology, any foreign matter from theregion-to-paint image acquired by the acquisition unit F11. Whendetermining whether or not there is any foreign matter, thedetermination unit F13 also determines the condition of the region topaint (including information about its hue and unevenness). Thecondition of the region to paint is reflected on the level of painting(coating level) by the painting system 40. For example, the amount ofthe paint dispensed (spray volume or spray concentration) is determinedby the level of painting (coating level). In this case, if no foreignmatter is detected (if the answer is NO in S33), then the determinationunit F13 starts painting (or coating) the region to paint (in S34).

To paint the regions to paint, the determination unit F13 controls thepainting system 40 to have the regions to paint (e.g., the hood and theright rear fender in the example illustrated in FIG. 13) painted by thepainting robots 41 and 42 as shown in FIG. 13. In addition, thedetermination unit F13 also determines the condition of the regionsbeing painted on the surface of the object 100. For this purpose, thedetermination unit F13 controls the image capturing system 30 to acquirean image of the regions to paint. In FIG. 13, the first cameras 311 and314 are used to acquire an image of the regions to paint. Thedetermination unit F13 obtains the difference between the paintingcondition of the surface (i.e., the current surface) derived from theimage representing the surface of the object 100 as acquired by theacquisition unit F11 (i.e., an image representing the region to paint)and the target painting condition on the surface of the object 100. Thedetermination unit F13 controls the painting system 40 so as to reducethe difference. For example, the determination unit F13 adjusts the spayconcentration (the amount of the paint ejected) of the paint dispensedfrom the painting robots 41 and 42. In the painting determinationprocessing, the determination unit F13 determines whether or not thereis any lack of coating in the region to paint (in terms of the hue anduniformity, for example). Optionally, in this painting determinationprocessing, the color of the surface of the object 100 may also bedetermined based on plurality of conditions of reflection having aspecular reflection component and a diffuse reflection component atmutually different ratios on the surface of the object 100, as in thecolor determination processing described above.

When the painting is completed, the determination unit F13 carries outpost-painting check (in S35). In the post-painting check, thedetermination unit F13 checks the condition after the painting. Examplesof the conditions to be checked after painting include whether or notthere is any paint curtaining, whether the paint is dry or not, and thedegree of continuity with the painted region (in terms of hue andsurface conditions). In this embodiment, the determination unit F13determines whether or not there is any paint curtaining in the regionbeing painted. For this purpose, the determination unit F13 controls theimage capturing system 30 to capture an image of the region that hasbeen painted using one of the plurality of first cameras 31 (e.g., thefirst camera 313 in FIG. 13) and acquire the image via the acquisitionunit F11. Based on the region-to-paint image acquired by the acquisitionunit F11, the determination unit F13 detects any paint curtaining. Thepaint curtaining may be detected based on the hue and surface condition(such as the color uniformity) of the region being painted, for example.The determination unit F13 obtains the difference between the paintingcondition of the surface (i.e., the current surface) derived from theimage representing the surface of the object 100 as acquired by theacquisition unit F11 (i.e., an image representing the region beingpainted) and the target painting condition on the surface of the object100. As the target painting condition on the surface of the object 100(i.e., a comparative image on which decision is made whether or notthere is any paint curtaining), a region painted uniformly with thepaint may be used out of the image representing the whole of the surfaceof the object 100 and captured by the second camera 32. When finding thedifference greater than a threshold value (i.e., a first thresholdvalue), the determination unit F13 may determine that there should besome paint curtaining Optionally, in determining whether or not there isany paint curtaining, the color of the surface of the object 100 mayalso be determined based on plurality of conditions of reflection havinga specular reflection component and a diffuse reflection component atmutually different ratios on the surface of the object 100, as in thecolor determination processing described above.

In this case, if no paint curtaining is detected (if the answer is NO inS36), then the determination unit F13 determines, based on the imagerepresenting the whole of the surface of the object 100 and captured bythe second camera 32, whether or not there is any unpainted region (inS371). If there is any unpainted region (if the answer is YES in S371),then the determination unit F13 determines where to paint next (in S31).On the other hand, if there are no unpainted regions (if the answer isNO in S371), then the determination unit F13 carries out a final check(in S372). In the final check, the determination unit F13 may performthe color determination processing to check the color of the whole ofthe surface of the object 100.

On the other hand, if any paint curtaining is detected (if the answer isYES in S36), then the determination unit F13 determines the degree ofpaint curtaining (in S381). More specifically, the determination unitF13 determines whether or not the degree of paint curtaining issignificant. The degree of paint curtaining may or may not besignificant depending on whether the paint curtaining may be repaired byre-painting. For example, when finding that the difference between theregion-to-paint image acquired by the acquisition unit F11 and thecomparative image has exceeded a second threshold value, which isgreater than a first threshold value, the determination unit F13 maydetermine that the paint curtaining should be significant. Optionally,the first threshold value and the second threshold value may be equal toeach other. Note that if any paint curtaining has been detected, thenthe determination unit F13 may generate paint curtaining recurrenceprevention information by associating the feature quantity of the imagerepresenting the paint curtaining with the amount of paint dispensed,room temperature, humidity, ventilation flow rate, or any otherparameter through statistical analysis or machine learning, for example.

On the other hand, when finding the degree of paint curtaininginsignificant (if the answer is NO in S381), the determination unit F13performs re-coating (re-painting) (in S382). In the re-coating, thedetermination unit F13 also controls the painting system 40 to paint theregion to paint, where the paint curtaining has been detected, all overagain as in the coating processing step (in S34). For example, FIG. 14illustrates how the first camera 314 captures an image of the trunk lidto have the trunk lid re-painted by the painting robot 42. When there-painting is completed, the determination unit F13 carries outpost-painting check again (in S35).

Note that if any foreign matter has been detected during the paintingprocessing (if the answer is YES in S33) or if the degree of paintcurtaining is significant (if the answer is YES in S381), then thedetermination unit F13 aborts the painting (coating) processing (inS39).

The painting processing includes painting determination processing (S34and S35). The painting determination processing is processing fordetermining the surface painting condition of the object 100. Morespecifically, the painting determination processing is processing fordetermining the surface painting condition of the object 100 based onthe image representing the surface of the object 100 as acquired by theacquisition unit F11. In the painting determination processing, thedetermination unit F13 obtains the difference between the surfacepainting condition derived from the image representing the surface ofthe object 100 as acquired by the acquisition unit F11 and the targetsurface painting condition for the object 100. The determination unitF13 controls, based on the result of the painting determinationprocessing, the operating conditions of the plurality of cameras 31 and32 of the image capturing system 30. That is to say, the determinationunit F13 controls the operating conditions of the plurality of cameras31 and 32 of the image capturing system 30 according to the progress ofpainting onto the object 100. In particular, the plurality of cameras 31and 32 includes one or more first cameras 31 each generating an imagerepresenting a part of the surface of the object 100 and a second camera32 generating an image representing the whole of the surface of theobject 100. The determination unit F13 controls the operating conditionsof the one or more first cameras 31 based on the result of the paintingdetermination processing. In addition, the determination unit F13 alsocontrols the operating conditions of the one or more first cameras 31based on the image captured by the second camera 32 and the result ofthe painting determination processing.

In this embodiment, the determination unit F13 controls the plurality ofcameras 31 and 32 of the image capturing system 30 to perform paintingon the object 100 in its entirety. In this case, if the object 100 is avehicle such as an automobile, then the object 100 has too large a sizeand has too many painting conditions to monitor (such as coating, paintcurtaining, drying, and foreign matter deposition) to be shot in itsentirety by a single camera. The inspection system 1 according to thisembodiment, however, makes the first cameras (narrow-angle cameras) 31for shooting only a local region of the object 100 being painted withpaint interface with a second camera (bird's-eye view camera) 32 forcapturing the painting condition on the entire object 100. This allowsthe inspection system 1 according to this embodiment to sense thepainting condition of the object 100 without missing the condition ofany part of the object 100.

(Texture Determination Processing)

The texture determination processing is processing for determining thesurface texture of the object 100. The texture determination processingis processing for determining the surface texture of the object 100based on a variation in luminance information between a plurality ofseries images acquired by capturing the images of the surface of theobject 100 from multiple different positions L1-L3 (see FIG. 16). Inthis embodiment, the variation in luminance information is defined byfeature quantity vectors (including a spatial feature quantity vectorand a temporal feature quantity vector).

Next, the texture determination processing will be described withreference to the flowchart of FIG. 15 and FIG. 16. FIG. 16 illustratesthe relative position of the second camera 32 with respect to the object100. Alternatively, the second camera 32 may be replaced with the firstcameras 31.

First of all, the determination unit F13 makes the acquisition unit F11acquire a plurality of series images which have been obtained bycapturing images of the surface of the object 100 from multipledifferent positions L1-L3 (see FIG. 16) (in S41). At least two of theplurality of series images are obtained by capturing images of thesurface of the object 100 with the position of the same camera 32changed. More specifically, a plurality of series images are obtained bymaking the camera 32 capture images of the surface of the object 100from multiple different positions L1, L2, L3, . . . as shown in FIG. 16.For example, these positions L1, L2, L3, . . . may be arranged so as tosurround the object 100 and may be arranged on a circle centered aroundthe object 100. Alternatively, the plurality of series images may alsobe captured by a plurality of cameras set up at those multiple differentpositions L1, L2, L3, . . . In the following description, only two outof the plurality of series images (hereinafter referred to as a “firstseries image” and a “second series image”) will be described. Note thatthe first series image is captured before the second series image.

Next, the determination unit F13 calculates the difference in luminancevalue between pixels (i.e., calculates a spatial feature quantityvector) (in S42). The determination unit F13 extracts luminanceinformation from the series images. The luminance information is thedifference between the luminance values (or pixel values) obtained froma plurality of pixels of the series images. The difference between theluminance values is the difference between the luminance value of afirst region including one or more of the plurality of pixels of theseries images and the luminance value of a second region adjacent to thefirst region and also including one or more of the plurality of pixels.For example, the first region may be a region consisting of m×n pixels,where m and n are each an integer equal to or greater than 1. In thisembodiment, the first region consists of 1×1 pixel (i.e., a singlepixel). The center pixel of the first region will be hereinafterreferred to as a “first pixel (reference pixel).” In this case, thefirst region is the brightest region in the series image (i.e., an imageconsisting of a plurality of pixels). In this example, the first regionconsists of the first pixel, and therefore, the first pixel is a pixelwith the highest luminance value among the plurality of luminance valuesof the series images. On the other hand, the second region may be aregion surrounding the first region. For example, the second region maybe a region consisting of M×N pixels centered around the first region,where m and n are each an integer equal to or greater than 3. In thisembodiment, the second region consists of all of the plurality of pixelsof the series image but the first pixel. That is to say, thedetermination unit F13 calculates the difference between the luminancevalue of the first pixel and the luminance value of each pixel for theseries image and replaces the luminance value of each pixel with thedifferential value. In this manner, the determination unit F13 obtains afeature quantity vector (spatial feature quantity vector) consisting ofthe luminance value of the first pixel and the replaced luminance values(differential values) of the plurality of pixels of the series image.Note that the first pixel does not have to be a pixel with the largestluminance value but may also be a pixel with the smallest luminancevalue, a pixel of which the luminance value is an average of the image,or a pixel located at the center of the image.

Next, the determination unit F13 calculates the difference in luminancevalue between frames (as a temporal feature quantity vector) (in S43).Specifically, the determination unit F13 calculates the differencebetween the luminance value of a first region of interest including oneor more of the plurality of pixels of a first series image and theluminance value of a second region of interest, corresponding to thefirst region of interest, among the plurality of pixels of a secondseries image. That is to say, the first region of interest and thesecond region of interest are selected so as to represent the same partof the surface of the object 100. For example, the first region ofinterest may be a region smaller than the first series image. The firstregion of interest nay be a region consisting of m×n pixels, where m andn are each an integer equal to or greater than 1. The center pixel ofthe first region of interest may be a pixel with the largest luminancevalue, a pixel with the smallest luminance value, or a pixel of whichthe luminance value is an average of the image. In this embodiment, thecenter pixel of the first region of interest is a pixel, of which theluminance value is an average of the image. The second region ofinterest is a region of the same size as the first region of interest(i.e., a region consisting of m×n pixels). The center pixel of thesecond region of interest is a pixel that has the smallest difference inluminance value from, and suitably has the same luminance value as, thecenter pixel of the first region of interest. Then, the determinationunit F13 calculates the difference between the respective luminancevalues of the first series image (first region of interest) and thesecond series image (second region of interest) and replaces theluminance value of the second series image (second region of interest)with the differential value. That is to say, the determination unit F13calculates the difference between the luminance value of the pixelincluded in the first region of interest of the first series image andthe luminance value of the pixel included in the second region ofinterest of the second series image. In this manner, the differentialvalues between the luminance values are obtained for the m×n pixels.Thus, the determination unit F13 obtains a feature quantity vector(i.e., a temporal feature quantity vector) including, as its elements,the luminance value of the center pixel of the first region of interestand the replaced pixel values (differential values) of the second seriesimage.

Finally, the determination unit F13 calculates the texture (texturelevel) (in S44). In this embodiment, the texture is given as acombination of the spatial feature quantity vector and the temporalfeature quantity vector. In other words, according to this embodiment,the surface texture of the object 100 is represented as a numericalvalue in the form of a combination of the spatial feature quantityvector and the temporal feature quantity vector. Then, the determinationunit F13 may determine, based on the numerical values, whether or notthe texture satisfies the required one. For example, the determinationunit F13 may determine, by seeing if the magnitude of the vectorindicating the texture is greater than a threshold value, whether or notthe texture satisfies the required one. When finding the texturesatisfying the required one, the determination unit F13 may determinethat the result of the texture test should be a GO. On the other hand,when finding the texture not satisfying the required one, thedetermination unit F13 may determine that the object 100 should beeither re-painted or the result of the texture test should be a NO-GO.

As can be seen, according to this embodiment, variations caused byrelative displacement between the camera 32 and the object 100 inluminance value are calculated along the spatial axis and the time axisand are integrated together. The variation in luminance value along thespatial axis (i.e., spatial variation) is given as the differencebetween the luminance value of the reference pixel and the luminancevalue of a neighboring pixel. Examples of the reference pixels includethe brightest pixel, the darkest pixel, a pixel with average brightness,and a pixel at the center of the image. The variation in luminance valuealong the time axis (temporal variation) is given as the differencebetween a reference frame and a neighboring frame. Thus, according tothis embodiment, the texture (i.e., a human touch sensation about thesurface condition) of the surface (in particular, a painted surface) ofthe object 100 is able to be measured and represented as a numericalvalue.

1.3 Resume

The inspection system 1 described above includes an acquisition unit F11and a determination unit F13. The acquisition unit F11 acquires an imagerepresenting the surface of an object 100. The determination unit F13performs color determination processing. The color determinationprocessing is performed to determine a color of the surface of theobject 100 based on a plurality of conditions of reflection. Theplurality of conditions of reflection are obtained from the imagerepresenting the surface of the object 100 as acquired by theacquisition unit F11, and have a specular reflection component and adiffuse reflection component at respectively different ratios on thesurface of the object 100. This allows the inspection system 1 toimprove the accuracy of surface color determination of the object 100.

In other words, it can be said that the inspection system 1 carries outthe following method (inspection method). The inspection method includesan acquisition step and a determination step. The acquisition stepincludes acquiring an image representing the surface of an object 100.The determination step includes performing color determinationprocessing. The color determination processing is performed to determinea color of the surface of the object 100 based on a plurality ofconditions of reflection. The plurality of conditions of reflection areobtained from the image representing the surface of the object 100 asacquired by the acquisition unit F11, and have a specular reflectioncomponent and a diffuse reflection component at respectively differentratios on the surface of the object 100. Thus, this inspection method,as well as the inspection system 1, improves the accuracy of surfacecolor determination of the object 100.

The inspection method is carried out by making one or more processorsexecute a program (computer program). This program is designed to makethe one or more processors carry out the inspection method. Such aprogram, as well as the inspection method, improves the accuracy ofsurface color determination of the object 100. Also, the program may bedistributed by being stored on a storage medium. The storage medium is acomputer readable, non-transitory storage medium, and stores the programthereon. Such a storage medium, as well as the inspection method,improves the accuracy of surface color determination of the object 100.

From another perspective, the inspection system 1 includes an imagecapturing system 30, an acquisition unit F11, and a determination unitF13. The image capturing system 30 generates an image representing thesurface of the object 100 by capturing an image of the surface of theobject 100. The acquisition unit F11 acquires the image representing thesurface of the object 100 from the image capturing system 30. Thedetermination unit F13 performs, based on the image representing thesurface of the object 100 as acquired by the acquisition unit F11,painting determination processing for determining a painting conditionon the surface of the object 100, and thereby controls the imagecapturing system 30 based on a result of the painting determinationprocessing. This inspection system 1 improves the quality of surfacepainting of the object 100.

From still another perspective, the inspection system 1 includes anacquisition unit F11 and a determination unit F13. The acquisition unitF11 acquires a plurality of series images that have been obtained bycapturing images of the surface of the object 100 from multipledifferent positions L1-L3. The determination unit F13 performs texturedetermination processing for determining a surface texture of the object100 based on a variation in luminance information between the pluralityof series images. This aspect further improves the accuracy of surfacetexture determination of the object (100).

2. Variations

Note that embodiments described above are only examples of the presentdisclosure and should not be construed as limiting. Rather, thoseembodiments may be readily modified in various manners depending on adesign choice or any other factor without departing from a true spiritand scope of the present invention. Variations of the embodiments willbe enumerated one after another.

In the exemplary embodiment described above, the cameras of the imagecapturing system 30 are able to detect light, of which the wavelengthfalls within a predetermined wavelength range, for example. Thepredetermined wavelength range may be from 380 nm to 780 nm, forexample. However, this is only an example and should not be construed aslimiting. Alternatively, the plurality of cameras of the image capturingsystem 30 may have filters with mutually different transmission bands.For example, the four first cameras 311-314 may be configured to detectlight rays with wavelengths falling within mutually different wavelengthranges. Such light rays with wavelengths falling within mutuallydifferent wavelength ranges may include a light ray with a wavelengthfalling within the range from 380 nm to 480 nm (i.e., a blue ray), alight ray with a wavelength falling within the range from 480 nm to 580nm (i.e., a green ray), a light ray with a wavelength falling within therange from 580 nm to 680 nm (i.e., a yellow ray), and a light ray with awavelength falling within the range from 680 nm to 780 nm (i.e., a redray) as shown in FIG. 17. Detecting light rays with wavelengths fallingwithin mutually different wavelength ranges using the plurality ofcameras 31 in this manner allows the color of the object 100 to bedetermined more precisely. Particularly in the example illustrated inFIG. 17, each of the four wavelength ranges is subdivided into ninebands at a step width of 10 nm, and therefore, the wavelength range from380 nm to 780 nm is detected by being divided into 36 bands. This allowsthe color of the object 100 to be determined more precisely than athree-band configuration, which is used relatively frequently.Optionally, the same advantage may be achieved by having a single camerause multiple filters with mutually different transmission bands.

In one variation, the wavelength of the light radiated from the lightingsystem 20 may be variable. This is achievable by using either multiplelight sources that emit light beams in multiple different colors ormultiple color filters. In short, in this inspection system 1, at leastone of the wavelength of the light radiated from the lighting system 20or the wavelength of the light detected by the image capturing system 3may be variable.

In the exemplary embodiment described above, the plurality of partialimages P31-P34 are generated by a plurality of cameras 31 havingmutually different image capturing directions with respect to the object100. However, this is only an example and should not be construed aslimiting. Alternatively, the plurality of partial images P31-P34 mayalso be obtained by capturing images of the surface of the object 100with the position of the same camera changed.

Also, in the exemplary embodiment described above, the plurality ofconditions of reflection having the specular reflection component andthe diffuse reflection component at mutually different ratios on thesurface of the object 100 are in the form of images. However, this isonly an example and should not be construed as limiting. Alternatively,the conditions of reflection may also be in the form of histograms orany other form. That is to say, the conditions of reflection from thesurface of the object 100 do not have to be given in the form of imagesbut may also be given in a form that enables color determinationaccording to the conditions of reflection.

In one variation, the setting processing does not have to be performed.Unless the setting processing is performed, the color determinationprocessing may be performed with sample data provided for each of theimages generated by the plurality of cameras 31. This allows theplurality of cameras 31 to perform the color determination processing onan individual basis on multiple different surface regions of the object100. Note that in that case, the images are not synthesized together atthe time of the color determination processing.

Furthermore, in the exemplary embodiment described above, thedetermination unit F13 controls, during the re-painting, the paintingsystem 40 based on the difference between the color obtained from thefirst synthetic image and the target color for the first synthetic imageand the difference between the color obtained from the second syntheticimage and the target color for the second synthetic image. However, thisis only an example and should not be construed as limiting.Alternatively, the determination unit F13 may control the paintingsystem 40 using learned models (color control models). As used herein,the “color control models” refer to learned models in which therelationship between a combination of a color yet to be modified and amodified color and the specifics of control for the painting system 40has been learned. In that case, the storage unit 12 stores the colorcontrol models. The color control models are generated by making anartificial intelligence program (algorithm) learn the relationshipbetween a combination of a color yet to be modified and a modified colorand the specifics of control for the painting system 40, using alearning data set representing the relationship between the combinationof the color yet to be modified and the modified color and the specificsof control for the painting system 40. The artificial intelligenceprogram is a machine learning model and may be a neural network, whichis a type of hierarchical model, for example. The color control modelmay be generated by making the neural network perform machine learning(such as deep learning) using the learning data set. That is to say, thecolor control model may be generated by either the processing unit 13 ofthe inspection system 1 or an external system. In the inspection system1, the processing unit 13 may collect and accumulate learning data forgenerating the color control models. As can be seen, the learning datanewly collected by the processing unit 13 may be used for relearning thecolor control models, thus contributing to performance improvement ofthe color control models (learned models). In particular, theperformance of the color control models may be improved throughrelearning in a situation where the result of the color determinationprocessing turns out to be a NO-GO again after the re-painting.

In one variation, the determination unit F13 may use models for thetexture determination processing. Those models may be obtained by makinga plurality of painting samples, making multiple pairs of GO/NO-GOdecisions of painting and the texture levels, and then modelling theirrelationship. Modelling may be carried out through either regressionanalysis or machine learning, for example. This allows the determinationunit F13 to make the GO/NO-GO decisions of painting based on the texturelevels. In another variation, only spatial feature quantity vectors maybe used to measure the texture levels when the positional relationbetween the camera 32 and the object 100 is fixed.

In another variation, in the texture determination processing, theluminance information may be the difference between the luminance valuesobtained from a plurality of pixels of the series image. This differencemay be the difference between the luminance value of a first regionincluding one or more of the plurality of pixels and the luminance valueof a second region adjacent to the first region and also including oneor more of the plurality of pixels. Alternatively, the first region mayalso be a first pixel out of the plurality of pixels and the secondregion may also be a second pixel adjacent to the first pixel out of theplurality of pixels. Still alternatively, the first region may be thebrightest region in an image made up of the plurality of pixels.

In still another variation, the inspection system 1 (determinationsystem 10) may also be implemented as a plurality of computers. Forexample, the respective functions (among other things, the acquisitionunit F11, the separation unit F12, and the determination unit F13) ofthe inspection system 1 (determination system 10) may be distributed inmultiple devices. Optionally, at least some functions of the inspectionsystem 1 (determination system 10) may be implemented as cloud computingas well.

The agent that performs the functions of the inspection system 1(determination system 10) described above includes a computer system. Inthat case, the computer system may include, as principal hardwarecomponents, a processor and a memory. The functions of the agent servingas the inspection system 1 (determination system 10) according to thepresent disclosure may be performed by making the processor execute aprogram stored in the memory of the computer system. The program may bestored in advance in the memory of the computer system. Alternatively,the program may also be downloaded through a telecommunications line orbe distributed after having been recorded in some non-transitory storagemedium such as a memory card, an optical disc, or a hard disk drive, anyof which is readable for the computer system. The processor of thecomputer system may be made up of a single or a plurality of electroniccircuits including a semiconductor integrated circuit (IC) or alargescale integrated circuit (LSI). Optionally, a field-programmablegate array (FPGA) to be programmed after an LSI has been fabricated or areconfigurable logic device allowing the connections or circuit sectionsinside of an LSI to be reconfigured or set up may also be used for thesame purpose. Those electronic circuits may be either integratedtogether on a single chip or distributed on multiple chips, whichever isappropriate. Those multiple chips may be integrated together in a singledevice or distributed in multiple devices without limitation.

3. Aspects

As can be seen from the foregoing description of exemplary embodimentsand variations, the present disclosure has the following aspects. In thefollowing description, reference signs are added in parentheses to therespective constituent elements, solely for the purpose of clarifyingthe correspondence between those aspects of the present disclosure andthe exemplary embodiments or variations described above.

A first aspect is an inspection system (1) including an acquisition unit(F11) and a determination unit (F13). The acquisition unit (F11)acquires an image (P30-P34) representing a surface of an object (100).The determination unit (F13) performs color determination processing.The color determination processing is performed to determine a color ofthe surface of the object (100) based on a plurality of conditions ofreflection. The plurality of conditions of reflection are obtained fromthe image (P30-P34) representing the surface of the object (100) asacquired by the acquisition unit (F11), and have a specular reflectioncomponent and a diffuse reflection component at respectively differentratios on the surface of the object (100). This aspect improves theaccuracy of surface color determination of the object (100).

A second aspect is based on the inspection system (1) according to thefirst aspect. In the second aspect, the inspection system (1) furtherincludes a separation unit (F12). The separation unit (F12) obtains,based on the image acquired by the acquisition unit (F11), a pluralityof separate images (P10, P20), each of which is an image (P30-P34)representing the surface of the object (100) but which have the specularreflection component and the diffuse reflection component atrespectively different ratios. The determination unit (F13) determines,through the color determination processing, the color of the surface ofthe object (100) based on the plurality of separate images (P10, P20).This aspect improves not only the accuracy of surface colordetermination of the object (100) but also the efficiency of the colordetermination processing as well.

A third aspect is based on the inspection system (1) according to thefirst or second aspect. In the third aspect, the acquisition unit (F11)acquires, as images representing the surface of the object (100), aplurality of partial images (P31-P34) each representing an associatedpart of the surface of the object (100). The determination unit (F13)determines, through the color determination processing, the color of thesurface of the object (100) based on an image (P10, P20) representing,in each of the plurality of conditions of reflection, a whole of thesurface of the object (100). The image (P10, P20) is obtained based onthe plurality of partial images. This aspect allows the color of thesurface of a relatively large object (100) to be determined.

A fourth aspect is based on the inspection system (1) according to thethird aspect. In the fourth aspect, the plurality of partial images(P31-P34) are generated by a plurality of cameras (31) having mutuallydifferent image capturing directions with respect to the object (100).This aspect allows the color of the surface of a relatively large object(100) to be determined using a simple configuration.

A fifth aspect is based on the inspection system (1) according to anyone of the first to fourth aspects. In the fifth aspect, the inspectionsystem (1) further includes a lighting system (20) and an imagecapturing system (30). The lighting system (20) irradiates the surfaceof the object (100) with light. The image capturing system (30)generates an image representing the surface of the object (100) bycapturing an image of the surface of the object (100) being irradiatedby the lighting system (20). The acquisition unit (F11) acquires, fromthe image capturing system (30), the image representing the surface ofthe object (100). At least one of a wavelength of the light radiated bythe lighting system (20) or a wavelength of light detected by the imagecapturing system (30) is changeable. This aspect improves the accuracyof surface color determination of the object (100).

A sixth aspect is based on the inspection system (1) according to anyone of the first to fifth aspects. In the sixth aspect, thedetermination unit (F13) determines, through the color determinationprocessing, the color of the surface of the object (100) using sampledata including information about a target color of the surface of theobject (100). This aspect further improves the accuracy of surface colordetermination of the object (100).

A seventh aspect is based on the inspection system (1) according to thesixth aspect. In the seventh aspect, the sample data includes at leastone of information about a shape of the object (100) or informationabout a condition for capturing an image of the object (100). Thisaspect further improves the accuracy of surface color determination ofthe object (100).

An eighth aspect is based on the inspection system (1) according to anyone of the first to seventh aspects. In the eighth aspect, thedetermination unit (F13) controls, based on a result of the colordetermination processing, a painting system (40) to paint the surface ofthe object (100). This aspect improves the quality of surface paintingof the object (100).

A ninth aspect is based on the inspection system (1) according to thefirst aspect. In the ninth aspect, the inspection system (1) furtherincludes an image capturing system (30) to generate an imagerepresenting the surface of the object (100) by capturing an image ofthe surface of the object (100). The acquisition unit (F11) acquires theimage representing the surface of the object (100) from the imagecapturing system (30). The determination unit (F13) performs, based onthe image representing the surface of the object (100) as acquired bythe acquisition unit (F11), painting determination processing fordetermining a painting condition on the surface of the object (100), andthereby controls the image capturing system (30) based on a result ofthe painting determination processing. This aspect improves the qualityof surface painting of the object (100).

A tenth aspect is based on the inspection system (1) according to theninth aspect. In the tenth aspect, the determination unit (F13)calculates, through the painting determination processing, a differencebetween a current painting condition on the surface of the object and atarget painting condition on the surface of the object (100). Thecurrent painting condition is obtained from an image representing thesurface of the object (100) as acquired by the acquisition unit (F11).This aspect improves the quality of surface painting of the object(100).

An eleventh aspect is based on the inspection system (1) according tothe ninth or tenth aspect. In the eleventh aspect, the image capturingsystem (30) includes a plurality of cameras (31, 32). The determinationunit (F13) controls, based on a result of the painting determinationprocessing, operating conditions of the plurality of cameras (31, 32) ofthe image capturing system (30). This aspect improves the quality ofsurface painting of the object (100).

A twelfth aspect is based on the inspection system (1) according to theeleventh aspect. In the twelfth aspect, the plurality of cameras (31,32) includes: one or more first cameras (31) to generate an imagerepresenting a part of the surface of the object (100); and a secondcamera (32) to generate an image representing the whole of the surfaceof the object (100). The determination unit (F13) controls, based on aresult of the painting determination processing, an operating conditionof the one or more first cameras (31). This aspect improves the qualityof surface painting of the object (100).

A thirteenth aspect is based on the inspection system (1) according tothe twelfth aspect. In the thirteenth aspect, the determination unit(F13) controls, based on the image generated by the second camera (32)and the result of the painting determination processing, the operatingcondition of the one or more first cameras (31). This aspect improvesthe quality of surface painting of the object (100).

A fourteenth aspect is based on the inspection system (1) according tothe first aspect. In the fourteenth aspect, the acquisition unit (F11)acquires a plurality of series images by capturing images of the surfaceof the object (100) from multiple different positions (L1-L3). Thedetermination unit (F13) performs texture determination processing fordetermining a surface texture of the object (100) based on a variationin luminance information between the plurality of series images. Thisaspect further improves the accuracy of surface texture determination ofthe object (100).

A fifteenth aspect is based on the inspection system (1) according tothe fourteenth aspect. In the fifteenth aspect, at least two of theplurality of series images are obtained by capturing images of thesurface of the object (100) with a position of the same camera changed.This aspect further improves the accuracy of surface texturedetermination of the object (100).

A sixteenth aspect is based on the inspection system (1) according tothe fourteenth or fifteenth aspect. In the sixteenth aspect, each of theplurality of series images includes a plurality of pixels. The luminanceinformation includes a difference between luminance values obtained fromthe plurality of pixels. This aspect further improves the accuracy ofsurface texture determination of the object (100).

A seventeenth aspect is based on the inspection system (1) according tothe sixteenth aspect. In the seventeenth aspect, the difference iscalculated between a luminance value of a first region including one ormore of the plurality of pixels and a luminance value of a second regionadjacent to the first region and including another one or more of theplurality of pixels. This aspect further improves the accuracy ofsurface texture determination of the object (100).

An eighteenth aspect is based on the inspection system (1) according tothe seventeenth aspect. In the eighteenth aspect, the first region isconstituted by a first pixel out of the plurality of pixels, and thesecond region is constituted by a second pixel adjacent to the firstpixel out of the plurality of pixels. This aspect further improves theaccuracy of surface texture determination of the object (100).

A nineteenth aspect is based on the inspection system (1) according tothe seventeenth or eighteenth aspect. In the nineteenth aspect, thefirst region is the brightest region in the image constituted of theplurality of pixels. This aspect further improves the accuracy ofsurface texture determination of the object (100).

A twentieth aspect is an inspection method including an acquisition stepand a determination step. The acquisition step includes acquiring animage (P30-P34) representing a surface of an object (100). Thedetermination step includes performing color determination processing.The color determination processing is performed to determine a color ofthe surface of the object (100) based on a plurality of conditions ofreflection. The plurality of conditions of reflection are obtained fromthe image (P30-P34) representing the surface of the object 100 asacquired by the acquisition unit (F11) and have a specular reflectioncomponent and a diffuse reflection component at respectively differentratios on the surface of the object (100). This aspect improves theaccuracy of surface color determination of the object (100). Note thatthe second through nineteenth aspects relating to the inspection system(1) are applicable in the form of an inspection method to this twentiethaspect.

A twenty-first aspect is a program designed to cause one or moreprocessors to execute the inspection method of the twentieth aspect.This aspect improves the accuracy of surface color determination of theobject (100).

A twenty-second aspect is a computer-readable non-transitory storagemedium storing the program of the twenty-second aspect thereon. Thisaspect improves the accuracy of surface color determination of theobject (100).

The present disclosure further has the following twenty-third tothirty-fourth aspects.

A twenty-third aspect is a coloring inspection device. The coloringinspection device includes a camera, of which a filter transmittinglight falling within a particular wavelength range is replaceable, andmakes color inspection on an object of shooting using an image capturedby the camera.

A twenty-fourth aspect is based on the coloring inspection deviceaccording to the twenty-third aspect. In the twenty-fourth aspect, thecoloring inspection device includes a plurality of the cameras, makes asynthetic image by synthesizing together images captured by theplurality of the cameras, and calculates and outputs an image capturingcondition based on the synthetic image.

A twenty-fifth aspect is based on the coloring inspection deviceaccording to the twenty-fourth aspect. In the twenty-fifth aspect, thefilters provided for the plurality of the cameras have mutuallydifferent transmission bands.

A twenty-sixth aspect is based on the coloring inspection deviceaccording to the twenty-fourth or twenty-fifth aspect. In thetwenty-sixth aspect, the plurality of the cameras shoot the object ofshooting from multiple different directions.

A twenty-seventh aspect is based on the coloring inspection deviceaccording to any one of the twenty-fourth to twenty-sixth aspects. Inthe twenty-seventh aspect, the coloring inspection device controls,based on the synthetic image, the angle of view and zooming of thecameras so as to maintain continuity of gradation of the object ofshooting.

A twenty-eighth aspect is based on the coloring inspection deviceaccording to the twenty-seventh aspect. In the twenty-eighth aspect, thecoloring inspection device controls the cameras based on informationabout the shape of the object of shooting and lighting information.

A twenty-ninth aspect is based on the coloring inspection deviceaccording to any one of the twenty-third to twenty-ninth aspects. In thetwenty-ninth aspect, the coloring inspection device records color dataas samples, compares the color of the object of shooting with the colordata as samples, and thereby controls a painting unit for painting theobject of shooting.

A thirtieth aspect is a coloring inspection method. The coloringinspection method includes: shooting an object of shooting from multipledifferent directions using a plurality of cameras provided with filterswith mutually different transmission bands; making a synthetic image bysynthesizing together images captured by the plurality of cameras; andperforming color inspection on the object of shooting based on thesynthetic image.

A thirty-first aspect is based on the coloring inspection methodaccording to the thirtieth aspect. In the thirty-first aspect, the colorinspection method includes outputting an image capturing condition basedon the synthetic image.

A thirty-second aspect is based on the coloring inspection methodaccording to the thirtieth or thirty-first aspect. In the thirty-secondaspect, the color inspection method includes controlling, based on thesynthetic image, the angle of view and zooming of the cameras so as tomaintain continuity of gradation of the object of shooting.

A thirty-third aspect is based on the coloring inspection deviceaccording to the thirty-first aspect. In the thirty-third aspect, thecoloring inspection device controls the cameras based on informationabout the shape of the object of shooting and lighting information.

A thirty-fourth aspect is based on the coloring inspection deviceaccording to any one of the thirtieth to thirty-third aspects. In thethirty-fourth aspect, the coloring inspection device records color dataas samples, compares the color of the object of shooting with the colordata as samples, and thereby controls a painting unit for painting theobject of shooting.

The present disclosure further has the following thirty-fifth tothirty-seventh aspects.

A thirty-fifth aspect is a system. The system includes: a paintinginformation acquisition unit (which means a group of cameras) foracquiring information about a painting condition at a certain point intime; a standard information retaining unit for retaining standardinformation indicating what the painting condition should be at thatpoint in time; and a control unit connected to the painting informationacquisition unit and the standard information retaining unit. Thecontrol unit calculates the difference between information provided bythe painting information acquisition unit and information provided bythe standard information retaining unit and transmits, based on thedifference, a control command to the painting information acquisitionunit (which means a group of cameras).

A thirty-sixth aspect is based on the system according to thethirty-fifth aspect. In the thirty-sixth aspect, the control command isgiven to change operating conditions (such as panning and zooming) ofthe cameras included in the painting information acquisition unit (whichmeans a group of cameras).

A thirty-seventh aspect is based on the system according to thethirty-sixth aspect. In the thirty-seventh aspect, the paintinginformation acquisition unit (which means a group of cameras) includes abird's-eye view camera and a narrow-angle camera. The control command isgiven to change operating conditions (such as panning and zooming) ofthe narrow-angle camera.

The present disclosure further has the following thirty-eighth toforty-first aspects.

A thirty-eighth aspect is an image capturing method. The image capturingmethod includes the steps of: displacing an image capture devicerelative to an object; and obtaining a variation in luminance valueinformation included in information collected by capturing images of theobject before and after the image capture device is displaced.

A thirty-ninth aspect is based on the image capturing method accordingto the thirty-eighth aspect. In the thirty-ninth aspect, the informationacquired by the image capture device before the image capture device isdisplaced is defined to be image information. The image informationincludes a plurality of pixels. The image capturing method includesobtaining a variation in difference between luminance values of theplurality of pixels before and after the image capture device isdisplaced.

A fortieth aspect is based on the image capturing method according tothe thirty-ninth aspect. In the fortieth aspect, the difference is adifference between a luminance value at a first pixel of the imageinformation and a luminance value at a second pixel adjacent to thefirst pixel.

A forty-first aspect is based on the image capturing method according tothe fortieth aspect. In the forty-first aspect, the first pixel is apixel that has the highest luminance value in the entire imageinformation.

The present application is based upon, and claims the benefit of foreignpriority to, U.S. Provisional Patent Application No. 62/596,247, filedon Dec. 8, 2017, U.S. Provisional Patent Application No. 62/699,935,filed on Jul. 18, 2018, and U.S. Provisional Patent Application No.62/699,942, filed on Jul. 18, 2018, the entire contents of which arehereby incorporated by reference.

REFERENCE SIGNS LIST

-   -   1 Inspection System    -   F11 Acquisition Unit    -   F12 Separation Unit    -   F13 Determination Unit    -   20 Lighting System    -   30 Image Capturing System    -   31 Camera (First Camera)    -   32 Camera (Second Camera)    -   40 Painting System    -   P10 First Separate Image (Separate Image)    -   P20 Second Separate Image (Separate Image)    -   P30 Image    -   P30-P34 Image (Partial Image)    -   L1-L3 Position    -   100 Object

1. An inspection system comprising: an acquisitor configured to acquirea plurality of images representing a surface of an object; a separatorconfigured to respectively obtain a plurality of first separate imagesfrom the plurality of images acquired by the acquisitor, the pluralityof first separate images representing a condition of reflection in whicha proportion of a specular reflection component on the surface of theobject is greater than a proportion of a diffuse reflection component onthe surface of the object; and a determiner configured to synthesizetogether the plurality of first separate images respectively obtainedfrom the plurality of images to generate a first synthetic image,wherein the determiner is configured to determine a condition of theobject based on the first synthetic image.
 2. The inspection system ofclaim 1, wherein the separator is configured to respectively obtain theplurality of first separate images and a plurality of second separateimages from the plurality of images acquired by the acquisitor.
 3. Theinspection system of claim 2, wherein the plurality of second separateimages represents a condition of reflection in which a proportion of adiffuse reflection component on the surface of the object is greaterthan a proportion of a specular reflection component on the surface ofthe object.
 4. The inspection system of claim 2, wherein the determineris configured to synthesize together the plurality of second separateimages respectively obtained from the plurality of images to generate asecond synthetic image, the determiner is configured to determine thecondition of the object based on the second synthetic image.
 5. Theinspection system of claim 2, further comprising: a lighting systemconfigured to irradiate the surface of the object with light, whereinthe lighting system includes a lamp for irradiating the surface of theobject with light, and the lamp radiates white light.
 6. The inspectionsystem of claim 3, wherein the determiner is configured to determine acolor of the surface of the object using sample data includinginformation about a target color of the surface of the object.
 7. Theinspection system of claim 3, wherein the determiner is configured todetermine a condition of the surface of the object using sample dataincluding information about a shape of the surface of the object.
 8. Aninspection method comprising the steps of: acquiring a plurality ofimages representing a surface of an object; respectively obtaining aplurality of separate images from the plurality of images acquired inthe step of acquiring the plurality of images, the plurality of separateimages representing a condition of reflection in which a proportion of aspecular reflection component on the surface of the object is greaterthan a proportion of a diffuse reflection component on the surface ofthe object; and synthesizing together the plurality of separate imagesrespectively obtained from the plurality of images to generate asynthetic image, wherein the step of synthesizing together the pluralityof separate images includes determining a condition of the object basedon the synthetic image.
 9. An inspection method comprising the steps of:acquiring a plurality of images representing a surface of an object;respectively obtaining a plurality of separate images from the pluralityof images acquired in the step of acquiring the plurality of images, theplurality of separate images representing a condition of reflection inwhich a proportion of a diffuse reflection component on the surface ofthe object is greater than a proportion of a specular reflectioncomponent on the surface of the object; and synthesizing together theplurality of separate images respectively obtained from the plurality ofimages to generate a synthetic image, wherein the step of synthesizingtogether the plurality of separate images includes determining acondition of the object based on the synthetic image.
 10. An inspectionsystem comprising: an acquisitor configured to acquire a plurality ofimages representing a surface of an object; and a storage configured tostore information about the surface of the object obtained based on theplurality of images acquired by the acquisitor.
 11. The inspectionsystem of claim 10, wherein the information about the surface of theobject includes information about a shape of the object.